The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
With the increasing volume and high complexity of data, data skew is a common occurrence in large scale Big Data processing. Since most Big Data frameworks leverage a synchronous distributed computing model, the data skew will significantly increase execution time of Big Data applications. In some serious cases, the applications cannot even finish because of the data skew. Furthermore, data distribution in a corporation varies every day, while most of the existing solutions cannot solve the data skew issue at run time.
Therefore, an unaddressed need exists in the art to address the aforementioned deficiencies and inadequacies.